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Be it known that we, John F. Ebersole, residing at 58 Cortland Drive, Bedford, NH 03110- 
4224, and John F. Ebersole, Jr., residing at 2 Saratoga Court, Bedford, NH 03 110, and Todd J. 
Furlong, residing at 14-B Apple Tree Drive, Goffstown, NH 03045, and all being citizens of the 
United States of America, have invented a certain new and useful: 

Augmented Reality Display Integrated with Self-Contained Breathing Apparatus 

of which the following is a specification. 
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Applicant: Ebersole et al. 

For: Augmented Reality Display Integrated with Self-Contained Breathing Apparatus 

CROSS REFERENCE TO RELATED APPLICATIONS 

of Provisional patent application 60/225,343 filed August 15, 
2000. 

GOVERNMENT RIGHTS 

This invention was made with Government support under Contract Number N61339-98-C- 
0036 awarded by the Department of the Navy. The Government has certain rights in the 
invention. 

FIELD OF THE INVENTION 

This invention relates to the fields of augmented reality and firefighter/emergency first 
responder training and operations. Specifically, the invention relates to an augmented reality 
display device to be used for firefighter or other emergency first responder training or other 
activities. 

COPYRIGHT INFORMATION 

A portion of the disclosure of this patent document contains material that is subject to 
copyright protection. The copyright owner has no objection to the facsimile reproduction by 
anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark 
Office records but otherwise reserves all copyright works whatsoever. 

BACKGROUND OF THE INVENTION 

Augmented Reality (AR) is emerging as a technology for assistance in performing a variety 
of tasks. AR systems can be used to allow firefighters and other emergency first responders to 
train using real equipment and virtual fire and smoke. As part of making the experience reaUstic 
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for firefighter trainees, it is important that the AR experience resemble the real experience as 
closely as possible. Therefore, it is important to integrate an AR system into real or mockup fire 
fighting equipment. 

SUMMARY OF THE INVENTION 

The purpose of the invention is to augment the experience of a user in a computer-enhanced 
fire-training scenario such that the user actually wears equipment that would be wom in a real 
fire. Specifically, the invention augments the experience by allowing the user to wear a 
firefighter's SCBA (Self-Contained Breathing Apparatus) while engaging in computer-enhanced 
fire training situations. Key aspects of the invention include (1) a real firefighter's SCBA, 
including catcher's mask- style straps, which gives the user the sensation of being in a real 
firefighting situation, (2) a head motion tracker, which is mounted on the SCBA to track the 
position and orientation of the firefighter's head, (3) a specially mounted camera and mirror 
configuration which takes a video image of what is directly in firont of the user's eyes, (4) a 
video display screen, which is used to project the computer-enhanced image the firefighter will 
see in fi-ont of his eyes, and (5) specially mounted head phones which can be used to project 
appropriate sounds into the firefighter's ears. 

The invention provides sufficient accuracy in real-time such that a computer can generate 
virtual images and mix them with the image data firom the specially mounted camera in a way 
that the user sees the virtual and real images mixed in real time. The head-mounted tracker 
allows a computer to synchronize the virtual and real images such that the user sees an image 
being updated correctly with his or her head. The headphones fiirther enhance the virtual/real 
experience by providing appropriate aural input. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG 1 is a diagram of the main components of the invention. 

FIG 2 is a diagram of a two-mirror layout for placing the camera viewpoint immediately in 
front of the wearer's eyes. 

FIG 3A is a diagram of a headphone attachment design. 

FIG 3B is a diagram of a bumper that can be used to protect a mirror. 
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FIG 4 is an exploded view of a mirror mount design that places minimum pressure on the 
mirror to minimize distortion due to compression. 

FIG 5A depicts a structure that can be used to protect a mirror from being bumped and to 
prevent the mirror mount from being hooked from underneath. 

FIG 5B is a cross-sectional view of the structure of FIG 5A. 

FIGS 6A-C are top, side, and end views, respectively, of the structure in FIG 5. 

FIGS 7A and 7B are top and side views, respectively, of a rugged mirror mount. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS OF THE INVENTION 

FIG 1 summarizes the main components of the invention. The major components of this 
invention are a video camera 3, a motion tracker 4, a head-moxmted display (HMD) 2, 
headphones 5, and a self-contained breathing apparatus (SCBA) mask 1 with a "catcher's mask" 
style straps 6 to hold the SCBA onto the wearer's head. 

Camera 

Any sufficiently lightweight video camera 3 can be used for this invention. A 
PANASONIC® (Matsushita Electric Corporation of America, One Panasonic Way, Secaucus, NJ 
07094) GP-KS162 micro ("lipstick" style) camera with a 7 mm lens (GP-LM7TA) is preferably 
used in the invention. Because the camera must be worn on the user's head, it should be 
lightweight and minimally obtrusive. The camera's field of view (FOV) must be close to that of 
the HMD for minimal distortion, Le,, to best map the image of the real world to the AR world; 
this also contributes to maximizing the user's perception of presence in the AR world. If the 
HMD is an optical see-through display, the camera is not required. However, the cun-ent 
preferred embodiment of the invention is a video-based AR display. 

The ideal location of a camera for minimal offset from the wearer's eyes is inside the user's 
eyes. Of course, this is impractical, so the optical path maybe folded using mirrors to allow the 
camera viewpoint to coincide with the wearer's eye location with the camera at some distance 
from the wearer's eyes. Using an even number of mirrors as in FIG 2 allows the camera images 



{H:\PA\CORP\16805\00035\A0483736.DOC} 



3 



to be used as-is, and using an odd number of mirrors requires the camera image to be flipped 
before it is displayed to the wearer of the display. 

Mounting of the mirror is a challenge because the mirror must be exposed, yet must also be 
protected. FIG 3B shows a bumper design for protecting the mirror and whatever the mirror may 
bump against. FIG 4 shows a mirror mount design that both protects the mirror and allows a 
minimal amount of pressure to be applied to the mirror, reducing the distortion that is created, 
especially with plastic mirrors. FIG 5 and FIG 6 show an angular frame structure to shield the 
mirror mount and prevent it from getting hooked from underneath. FIG. 7 shows an alternative 
rugged mirror mount with a raised area at the front to prevent the mirror from getting bumped 
from that direction. 

For a stereoscopic embodiment of the invention, two cameras are required. They should be 
spaced the same distance apart as the wearer's eyes and possibly have an interpupillary distance 
(IPD) adjustment. Each camera can be mounted similarly to the way a single camera mount is 
described above. 

Motion Tracker 

Motion tracking equipment 4 is used to provide real-time, 6 degree of freedom (DOF), 
position and orientation information about a tracked object. By attaching a motion tracker 
rigidly to the camera, the camera can be tracked. Knowledge of the camera field of view and its 
position and orientation allows a computer to overlay computer-generated images on the camera 
video that appear to be anchored to locations in 3-D space. Altematively, attaching the tracker to 
a user's head allows the user's eye positions to be tracked, enabling the see-through embodiment 
of this technology. Any tracker that provides suitable 6 DOF measurements can be used as a 
part of this invention. Example technologies for motion tracking include magnetic, acousto- 
inertial, and optical. Two preferred trackers for this invention are the INTERSENSE 
(InterSense, Inc., 73 Second Avenue, Burlington, MA 01803, USA) IS-900™ and the 
INTERSENSE (InterSense, Inc., 73 Second Avenue, Burlington, MA 01803, USA) IS-600 Mark 
2 Plus™. 
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HMD 

The HMD 2, as stated above, can be either see-through or non-see-through in this invention. 
As most HMDs are too large to fit inside an SCBA mask, the preferred method of attachment to 
the SCBA mask is to cut out part or all of the transparent (viewing) portion of the mask to allow 
the bulk of the HMD to stick out, while placing the HMD optics close to the wearer's eyes. 
Holes drilled through the mask provide attachment points for the HMD. By drilling through the 
transparent portion of the mask, a generic solution can be achieved that does not depend upon the 
geometry of any particular SCBA mask or HMD. The preferred HMD for this invention is the 
VIRTUAL RESEARCH (Virtual Research Systems, Lie, 3824 Vienna Drive, Aptos , California 
95003) V6™ for a non-see-through method. 

Headphones 

Headphones 5 must be attached to the SCBA if audio is part of the AR application. Two 
requirements for the headphones are that they should not block out real- world sounds, and they 
should not interfere with dorming the mask or other firefighter equipment. To accomplish these 
purposes, a pair of headphones (AIWA [AIWA AMERICA, INC., 800 Corporate Drive 
Mahwah, NJ 07430] HP-A091 Stereo Headphones'^'^) rigidly mounted to the SCBA mask at a 
distance from the wearer's ears can be used. See FIG 3. Additional strength can be added to the 
shafts that connect the headphones to the SCBA by means of hardened epoxy resin, which can be 
accomphshed with JB WELD™ (JB Weld Company, P.O. Box 483, Sulphur Springs, TX 
75483). 

SCBA Mask 

Any SCBA mask 1 can be used with this invention. One preferred mask is the SCOTT (Scott 
Aviation, A Scott Technologies Company, Erie. Lancaster, NY 14086) AV2000™. This mask 
is an example of the state of the art for firefighting equipment, and the design of the mask has a 
hole near the wearer's mouth that allows easy breathing and speaking when a regulator is not 
attached. When performing the integration of the invention, the mask face seal, the "catcher's 
mask" style straps for attaching the mask, and the nose cup are features that are preserved. 

When integrating an AR display, it is also necessary to black out any view beyond the 
augmented reality portions of the user's field of view to ensure that the trainee's only view of the 
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outside world is the AR view displayed to him/her by the HMD. In the case where an SCBA is 
used, the rest of the SCBA can be blacked out by using an opaque substance such as tape, foam, 
plastic, rubber, silicone, paint, or preferably a combination of plastic, silicone, and paint. This is 
done to ensure that the trainee doesn't see the un-augmented real world by using his or her un- 
augmented peripheral vision to see around AR smoke or other artificial (computer-generated 
virtual) obstacles. 

Although specific features of the invention are shown in the drawing and not others, this is 
for convenience only, as each feature may be combined with any or all of the other features in 
accordance with the invention. 

Other embodiments that will occur to those skilled in the art are within the following claims. 

What is claimed is: 
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